The cancellation came as trade talks were held in Washington and US President Donald Trump said he wanted a complete trade deal with the Asian nation, not just an agreement for China to buy more US agricultural goods.
"It's trade related and markets are just hyper-sensitive to trade," said Paul Nolte, portfolio manager at Kingsview Asset Management in Chicago.
Recent signs of easing trade worries had supported the markets, bringing the S&P 500 just shy of its all-time high hit in July. Now it stands 1.2% away from that level.
Seven of the 11 major S&P sectors were lower with tariff-sensitive technology stocks losing the most, down 1.27%. The Philadelphia chip index fell 1.67%.
At 1:49 p.m. ET, the Dow Jones Industrial Average was down 128.44 points, or 0.47%, at 26,966.35, the S&P 500 was down 15.33 points, or 0.51%, at 2,991.46. The Nasdaq Composite was down 82.67 points, or 1.01%, at 8,100.21.
Before news of the farm visit cancellation broke, the S&P 500 and Dow Industrials were edging higher, supported by gains in healthcare stocks, while a drop in shares of Netflix Inc kept the Nasdaq firmly in the red.
Netflix slipped 7% after CEO Reed Hastings' comments underscored growing costs and rising competition from Walt Disney Co, Apple Inc and other video streaming services. Roku Inc tumbled 21.7% after Pivotal Research started coverage of its shares with a "sell" rating.